Searching long egocentric videos with natural language queries (NLQ) has compelling applications in augmented reality and robotics, where a fluid index into everything that a person (agent) has seen before could augment human memory and surface relevant information on demand. However, the structured nature of the learning problem (free-form text query inputs, localized video temporal window outputs) and its needle-in-a-haystack nature makes it both technically challenging and expensive to supervise. We introduce Narrations-as-Queries (NaQ), a data augmentation strategy that transforms standard video-text narrations into training data for a video query localization model. Validating our idea on the Ego4D benchmark, we find it has tremendous impact in practice. NaQ improves multiple top models by substantial margins (even doubling their accuracy), and yields the very best results to date on the Ego4D NLQ challenge, soundly outperforming all challenge winners in the CVPR and ECCV 2022 competitions and topping the current public leaderboard. Beyond achieving the state-of-the-art for NLQ, we also demonstrate unique properties of our approach such as gains on long-tail object queries, and the ability to perform zero-shot and few-shot NLQ.

Different video understanding tasks are typically treated in isolation, and even with distinct types of curated data (e.g., classifying sports in one dataset, tracking animals in another). However, in wearable cameras, the immersive egocentric perspective of a person engaging with the world around them presents an interconnected web of video understanding tasks -- hand-object manipulations, navigation in the space, or human-human interactions -- that unfold continuously, driven by the person's goals. We argue that this calls for a much more unified approach. We propose EgoTask Translation (EgoT2), which takes a collection of models optimized on separate tasks and learns to translate their outputs for improved performance on any or all of them at once. Unlike traditional transfer or multi-task learning, EgoT2's flipped design entails separate task-specific backbones and a task translator shared across all tasks, which captures synergies between even heterogeneous tasks and mitigates task competition. Demonstrating our model on a wide array of video tasks from Ego4D, we show its advantages over existing transfer paradigms and achieve top-ranked results on four of the Ego4D 2022 benchmark challenges.

While object reconstruction has made great strides in recent years, current methods typically require densely captured images and/or known camera poses, and generalize poorly to novel object categories. To step toward object reconstruction in the wild, this work explores reconstructing general real-world objects from a few images without known camera poses or object categories. The crux of our work is solving two fundamental 3D vision problems -- shape reconstruction and pose estimation -- in a unified approach. Our approach captures the synergies of these two problems: reliable camera pose estimation gives rise to accurate shape reconstruction, and the accurate reconstruction, in turn, induces robust correspondence between different views and facilitates pose estimation. Our method FORGE predicts 3D features from each view and leverages them in conjunction with the input images to establish cross-view correspondence for estimating relative camera poses. The 3D features are then transformed by the estimated poses into a shared space and are fused into a neural radiance field. The reconstruction results are rendered by volume rendering techniques, enabling us to train the model without 3D shape ground-truth. Our experiments show that FORGE reliably reconstructs objects from five views. Our pose estimation method outperforms existing ones by a large margin. The reconstruction results under predicted poses are comparable to the ones using ground-truth poses. The performance on novel testing categories matches the results on categories seen during training. Project page: https://ut-austin-rpl.github.io/FORGE/

We present a retrospective on the state of Embodied AI research. Our analysis focuses on 13 challenges presented at the Embodied AI Workshop at CVPR. These challenges are grouped into three themes: (1) visual navigation, (2) rearrangement, and (3) embodied vision-and-language. We discuss the dominant datasets within each theme, evaluation metrics for the challenges, and the performance of state-of-the-art models. We highlight commonalities between top approaches to the challenges and identify potential future directions for Embodied AI research.

第一人称视频在其持续环境的背景下突出了摄影师的活动。但是，当前的视频理解方法是从短视频剪辑中的视觉特征的原因，这些视频片段与基础物理空间分离，只捕获直接看到的东西。我们提出了一种方法，该方法通过学习摄影师（潜在看不见的）本地环境来促进以人为中心的环境的了解来链接以自我为中心的视频和摄像机随着时间的推移而张开。我们使用来自模拟的3D环境中的代理商的视频进行训练，在该环境中，环境完全可以观察到，并在看不见的环境的房屋旅行的真实视频中对其进行测试。我们表明，通过将视频接地在其物理环境中，我们的模型超过了传统的场景分类模型，可以预测摄影师所处的哪个房间（其中帧级信息不足），并且可以利用这种基础来定位与环境相对应的视频瞬间 - 中心查询，优于先验方法。项目页面：http：//vision.cs.utexas.edu/projects/ego-scene-context/

我们介绍了Soundspaces 2.0，这是一个用于3D环境的基于几何的音频渲染的平台。考虑到现实世界环境的3D网格，Soundspaces可以为从任意麦克风位置捕获的任意声音生成高度逼真的声音。它与现有的3D视觉资产一起支持一系列视听研究任务，例如视听导航，映射，源定位和分离以及声学匹配。与现有资源相比，Soundspaces 2.0具有允许连续的空间采样，对新型环境的概括以及可配置的麦克风和材料属性的优点。据我们所知，这是第一个基于几何的声学模拟，它提供了高忠诚和现实主义，同时也足够快地用于体现学习。我们展示了模拟器的属性，并根据现实世界的音频测量进行了基准性能。此外，通过涵盖具体导航和远场自动语音识别的两个下游任务，突出了后者的SIM2REAL性能。 Soundspaces 2.0可公开使用，以促进对感知系统的更广泛研究，这些系统既可以看到和听到。

房间冲动响应（RIR）函数捕获周围的物理环境如何改变听众听到的声音，对AR，VR和机器人技术中的各种应用产生影响。估计RIR的传统方法在整个环境中采用密集的几何形状和/或声音测量值，但我们探讨了如何根据空间中观察到的一组稀疏图像和回声来推断RIR。为了实现这一目标，我们介绍了一种基于变压器的方法，该方法使用自我注意力来构建丰富的声学环境，然后通过跨注意来预测任意查询源接收器位置的河流。此外，我们设计了一个新颖的训练目标，该目标改善了RIR预测与目标之间的声学​​特征中的匹配。在使用3D环境的最先进的视听模拟器的实验中，我们证明了我们的方法成功地生成了任意RIR，优于最先进的方法，并且在与传统方法的主要背离中 - 以几种方式概括新的环境。项目：http：//vision.cs.utexas.edu/projects/fs_rir。

我们介绍了视觉匹配任务，其中音频剪辑被转换为听起来像是在目标环境中记录的。鉴于目标环境的图像和源音频的波形，目标是重新合成音频，以匹配目标室声音的可见几何形状和材料所建议的。为了解决这一新颖的任务，我们提出了一个跨模式变压器模型，该模型使用视听注意力将视觉属性注入音频并生成真实的音频输出。此外，我们设计了一个自我监督的训练目标，尽管他们缺乏声学上不匹配的音频，但可以从野外网络视频中学习声学匹配。我们证明，我们的方法成功地将人类的言语转化为图像中描绘的各种现实环境，表现优于传统的声学匹配和更严格的监督基线。

我们探索动态声源的主动音频分离，其中体现的代理在3D环境中智能移动，以连续隔离感兴趣的对象发出的随时间变化的音频流。该经纪人听到了多种音频来源的混杂流（例如，在嘈杂的派对上演奏音乐和乐队的乐队）。考虑到有限的时间预算，它需要使用以自我为中心的视听观察来准确地提取目标声音。我们提出了一种配备新型变压器记忆的增强式学习代理，该学习者学习运动策略，以控制其相机和麦克风以恢复动态目标音频，并使用自我意见来对当前时间段进行高质量的估计，并同时改善其过去的估计。使用在现实世界扫描的Matterport3D环境中使用高度现实的声音空间模拟，我们表明我们的模型能够学习有效的行为，以进行动态音频目标的连续分离。项目：https：//vision.cs.utexas.edu/projects/active-av-dynamic-separation/。

对象目标导航的最新方法依赖于增强学习，通常需要大量的计算资源和学习时间。我们提出了使用无互动学习（PONI）的对象导航的潜在功能，这是一种模块化方法，可以散布“在哪里看？”的技能？对于对象和“如何导航到（x，y）？”。我们的主要见解是“在哪里看？”可以纯粹将其视为感知问题，而没有环境相互作用就可以学习。为了解决这个问题，我们提出了一个网络，该网络可以预测两个在语义图上的互补电位功能，并使用它们来决定在哪里寻找看不见的对象。我们使用在自上而下的语义图的被动数据集上使用受监督的学习来训练潜在的功能网络，并将其集成到模块化框架中以执行对象目标导航。 Gibson和MatterPort3D的实验表明，我们的方法可实现对象目标导航的最新方法，同时减少培训计算成本高达1,600倍。可以使用代码和预训练的模型：https：//vision.cs.utexas.edu/projects/poni/